Communication control device, communication control method, and program

ABSTRACT

There is provided a communication control device including: a data storage unit storing feature data representing features of appearances of one or more communication devices; an environment map building unit for building an environment map representing positions of communication devices present in a real space based on an input image obtained by imaging the real space and the feature data stored in the data storage unit; a detecting unit for detecting a user input toward a first communication device designating any data provided in the first communication device and a direction; a selecting unit for selecting a second communication device serving as a transmission destination of the designated data from the environment map based on the direction designated by the user input; and a communication control unit for transmitting the data provided in the first communication device from the first communication device to the second communication device.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a communication control device, acommunication control method, and a program.

2. Description of the Related Art

Recently, technology called augmented reality (AR), which processes animage obtained by imaging a real space and presents the processed imageto a user, has been receiving attention. In the AR technology, it isimportant that a computer recognize a situation of the real space. Forexample, as technology for recognizing the situation of the real space,Japanese Patent Laid-Open Publication No. 2008-304268 discloses a methodof dynamically generating an environment map representingthree-dimensional positions of physical objects present in the realspace by applying technology called simultaneous localization andmapping (SLAM) capable of simultaneously estimating a position and aposture of a camera and positions of feature points present in an imageof the camera. A basic principle of the SLAM technology using amonocular camera is disclosed in “Real-Time Simultaneous Localizationand Mapping with a Single Camera” (Andrew J. Davison, Proceedings of the9th IEEE International Conference on Computer Vision Volume 2, 2003, pp.1403-1410).

Incidentally, information communication technology is widely usedbetween general users at present, and many users have a plurality ofcommunication devices communicable with each other. For example, printdata transmission from a personal computer (PC) to a printer, image datatransmission from a digital camera to a PC, a data exchange between adesktop PC and a mobile phone, a connection data exchange betweenwireless devices, and the like are all communication betweencommunication devices communicable with each other. For example,technology for improving convenience of users upon communication betweenthe communication devices is disclosed in Japanese Patent Laid-OpenPublication No. 2001-142825.

SUMMARY OF THE INVENTION

In general, when data is exchanged between the above-describedcommunication devices, the user designates desired transmission data anda device of a communication partner on a screen of any one of atransmission source and a transmission destination, and issues a datatransmission command. As a method of issuing the data transmissioncommand, there are various types such as an icon dragging, menuselection, a text-based command input, and the like. However, the usershould know an address, an identification name, or an icon representingthe communication partner device in any method. This makes user'soperation complicated and prevents the user from performing an intuitivemanipulation. “Pick and Drop” disclosed in the above-described JapanesePatent Laid-Open Publication No. 2001-142825 is a method directed to adata exchange between devices with an intuitive user interface. However,since a special interface should be prepared in the devices of both thetransmission source and the transmission destination for the “Pick andDrop” method, the user may not enjoy sufficient convenience.

Meanwhile, the above-described environment map enables a computer toaccurately recognize a position of a communication device located in areal space. Thus, it is expected that a data transmission instructioncan be more intuitively provided between devices by a user interface towhich the environment map is applied.

In light of foregoing, it is desirable to provide a novel and improvedcommunication device, a communication control method, and a program thatprovide an intuitive user interface for data transmission betweendevices by applying an environment map.

According to an embodiment of the present invention, there is provided acommunication control device including: a data storage unit storingfeature data representing features of appearances of one or morecommunication devices; an environment map building unit for building anenvironment map representing positions of communication devices presentin a real space on the basis of an input image obtained by imaging thereal space and the feature data stored in the data storage unit; adetecting unit for detecting a user input toward a first communicationdevice designating any data provided in the first communication deviceand a direction; a selecting unit for selecting a second communicationdevice serving as a transmission destination of the designated data fromthe environment map on the basis of the direction designated by the userinput; and a communication control unit for transmitting the dataprovided in the first communication device from the first communicationdevice to the second communication device.

According to this configuration, the environment map representing thepositions of the communication devices located in the real space isdynamically built on the basis of the input image obtained by imagingthe real space. When the user input toward the first communicationdevice of a data transmission source is detected, designated data istransmitted from the first communication device to the secondcommunication device selected from the environment map on the basis ofthe direction designated by the user input.

The selecting unit may select a communication device located in thedirection designated by the user input with respect to the firstcommunication device in the environment map as the second communicationdevice.

The selecting unit may specify one straight line having a start point atthe first communication device on the basis of a position and a postureof the first communication device in the environment map and thedirection designated by the user input, and select a communicationdevice located at least near the straight line as the secondcommunication device.

If a plurality of communication devices are located in the directiondesignated by the user input with respect to the first communicationdevice in the environment map, the selecting unit may select acommunication device located nearest the first communication deviceamong the plurality of communication devices as the second communicationdevice.

The communication control device may further include a display controlunit for causing a display device to display an animation representingdata transmission when the data is transmitted from the firstcommunication device to the second communication device.

The user input may be dragging an icon displayed on a screen provided inthe first communication device and the data to be transmitted from thefirst communication device may be specified according to which icon isdragged by a user.

According to another embodiment of the present invention, there isprovided a communication control method by a communication controldevice including a storage medium storing feature data representingfeatures of appearances of one or more communication devices, includingthe steps of: building an environment map representing positions ofcommunication devices present in a real space on the basis of an inputimage obtained by imaging the real space and the feature data; detectinga user input toward a first communication device designating any dataprovided in the first communication device and a direction; selecting asecond communication device serving as a transmission destination of thedesignated data from the environment map on the basis of the directiondesignated by the user input; and transmitting the data provided in thefirst communication device from the first communication device to thesecond communication device.

According to another embodiment of the present invention, there isprovided a program for causing a computer, which controls acommunication control device including a storage medium storing featuredata representing features of appearances of one or more communicationdevices, to function as: an environment map building unit for buildingan environment map representing positions of communication devicespresent in a real space on the basis of an input image obtained byimaging the real space and the feature data stored in the data storageunit; a detecting unit for detecting a user input toward a firstcommunication device designating any data provided in the firstcommunication device and a direction; a selecting unit for selecting asecond communication device serving as a transmission destination of thedesignated data from the environment map on the basis of the directiondesignated by the user input; and a communication control unit fortransmitting the data provided in the first communication device fromthe first communication device to the second communication device.

According to another embodiment of the present invention, there isprovided a communication control device including: a data storage unitstoring feature data representing features of appearances of one or morecommunication devices; an environment map building unit for building anenvironment map representing positions of communication devices presentin a real space on the basis of an input image obtained by imaging thereal space and the feature data stored in the data storage unit; adetecting unit for detecting a user input toward a first communicationdevice designating any data provided in the first communication device;a selecting unit for selecting a second communication device serving asa transmission destination of the designated data from the environmentmap on the basis of a position of the first communication device in theenvironment map; and a communication control unit for transmitting thedata provided in the first communication device from the firstcommunication device to the second communication device.

The selecting unit may select a communication device located below thefirst communication device in the environment map as the secondcommunication device.

According to another embodiment of the present invention, there isprovided a communication control method by a communication controldevice including a storage medium storing feature data representingfeatures of appearances of one or more communication devices, includingthe steps of: building an environment map representing positions ofcommunication devices present in a real space on the basis of an inputimage obtained by imaging the real space and the feature data; detectinga user input toward a first communication device designating any dataprovided in the first communication device; selecting a secondcommunication device serving as a transmission destination of thedesignated data from the environment map on the basis of a position ofthe first communication device in the environment map; and transmittingthe data provided in the first communication device from the firstcommunication device to the second communication device.

According to another embodiment of the present invention, there isprovided a program for causing a computer, which controls acommunication control device including a storage medium storing featuredata representing features of appearances of one or more communicationdevices, to function as: an environment map building unit for buildingan environment map representing positions of communication devicespresent in a real space on the basis of an input image obtained byimaging the real space and the feature data stored in the data storageunit; a detecting unit for detecting a user input toward a firstcommunication device designating any data provided in a firstcommunication device; a selecting unit for selecting a secondcommunication device serving as a transmission destination of thedesignated data from the environment map on the basis of a position ofthe first communication device in the environment map; and acommunication control unit for transmitting the data provided in thefirst communication device from the first communication device to thesecond communication device.

According to the communication control device, the communication controlmethod, and the program according to embodiments of the presentinvention as described above, an intuitive user interface for datatransmission between devices can be provided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an overview of a systemaccording to an embodiment;

FIG. 2 is an illustrative diagram showing an example of an input imageacquired in a communication control device according to an embodiment;

FIG. 3 is a block diagram showing an example of a configuration of acommunication device according to an embodiment;

FIG. 4 is a block diagram showing an example of a configuration of thecommunication control device according to an embodiment;

FIG. 5 is a flowchart showing an example of a flow of a self-positiondetection process according to an embodiment;

FIG. 6 is an illustrative diagram illustrating a feature point set on anobject;

FIG. 7 is an illustrative diagram illustrating the addition of a featurepoint;

FIG. 8 is an illustrative diagram illustrating an example of aprediction model;

FIG. 9 is an illustrative diagram illustrating an example of aconfiguration of feature data;

FIG. 10 is a flowchart showing an example of a flow of an objectrecognition process according to an embodiment;

FIG. 11 is an illustrative diagram illustrating a user input to thecommunication device according to an embodiment;

FIG. 12 is a first illustrative diagram illustrating the selection of atransmission destination according to an embodiment;

FIG. 13 is a second illustrative diagram illustrating the selection of atransmission destination according to an embodiment;

FIG. 14 is a flowchart showing an example of a transmission destinationselection process according to an embodiment;

FIG. 15 is an illustrative diagram showing an example of an animationrelated to data transmission according to an embodiment;

FIG. 16 is a flowchart showing an example of a flow of a communicationcontrol process according to an embodiment;

FIG. 17 is an illustrative diagram illustrating the selection of atransmission destination according to a modified example;

FIG. 18 is a flowchart showing an example of a flow of a transmissiondestination selection process according to a modified example.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

The “detailed description of the embodiment(s)” will be given in thefollowing order.

1. Overview of System

2. Configuration of Communication Device according to Embodiment

3. Configuration of Communication Control Device according to Embodiment

-   -   3-1. Imaging Unit    -   3-2. Environment Map Generating Unit    -   3-3. Communication Processing Unit    -   3-4. Display Control Unit    -   3-5. Flow of Communication Control Process

4. Modified Examples

5. Summary

1. OVERVIEW OF SYSTEM

First, the overview of the system according to an embodiment of thepresent invention will be described with reference to FIGS. 1 and 2.FIG. 1 is a schematic diagram illustrating the overview of the systemaccording to the embodiment of the present invention. FIG. 2 is anillustrative diagram showing an example of an input image capable ofbeing acquired in a communication control device 100 according to theembodiment illustrated in FIG. 1.

Referring to FIG. 1, an environment 1 is shown as an example in whichthe communication control device 100 may be used. A table 10 andcommunication devices 20 a, 20 b, 20 c, and 20 d are present inside theenvironment 1. The table 10 is installed on a floor, which is the bottomof the environment 1. For example, the communication device 20 acorresponds to a mobile phone such as a smart phone or the like, and isheld by a user. For example, the communication device 20 b correspondsto a notebook PC, and is located on the table 10. For example, thecommunication device 20 c corresponds to a digital camera, and islocated on the table 10. For example, the communication device 20 dcorresponds to a printer, and is installed on the floor. Thecommunication devices 20 a, 20 b, 20 c, and 20 d can communicate witheach other via a wired or wireless communication connection.

The communication control device 100 is a communication device having acamera and a head mounted display (HMD) 104 mounted on a user Ua. A mainbody of the communication control device 100 may not necessarily bemounted on the user Ua. The communication control device 100 cancommunicate with the communication devices 20 a, 20 b, 20 c, and 20 dvia a wired or wireless communication connection. The communicationcontrol device 100 images the environment 1 as an example shown in FIG.1 and acquires a set of input images. The communication control device100 builds an environment map to be described later on the basis of theacquired input images. The communication control device 100 detects auser input related to data transmission to any one of the communicationdevices 20 a, 20 b, 20 c, and 20 d. The communication control device 100controls data transmission from one device to another device on thebasis of the detected user input.

FIG. 2 is an illustrative view showing an input image 106 as an exampleacquired by the communication control device 100. Referring to FIG. 2,the communication devices 20 a and 20 d shown in FIG. 1 are shown in theinput image 106. Two icons corresponding to data provided in thecommunication device 20 a are displayed on a screen of the communicationdevice 20 a. In this embodiment, for example, data is transmitted fromthe communication device 20 a to another communication device undercontrol of the communication control device 100 by the user Uaperforming a user input to be described later on the icon.

In this specification, the communication devices 20 a, 20 b, 20 c, and20 d are collectively referred to as the communication devices 20 byomitting letters of their reference symbols when there is no particularneed to distinguish among them. The same applies to other elements.

2. CONFIGURATION OF COMMUNICATION DEVICE ACCORDING TO EMBODIMENT

FIG. 3 is a block diagram showing an example of a configuration of thecommunication device 20 according to an embodiment of the presentinvention. Referring to FIG. 3, the communication device 20 includes astorage unit 40, a display unit 42, a user interface (I/F) 44, acommunication I/F 46, and a control unit 48.

The storage unit 40 stores data using a storage medium such as a harddisk or a semiconductor memory. The data stored by the storage unit 40may be any type of data such as application data, text data, image data,audio data, or program data.

The display unit 42 displays information on a screen provided on thecommunication device 20 according to control by the control unit 48. Forexample, as illustrated in FIG. 2, the display unit 42 displays iconsrespectively corresponding to data stored in the storage unit 40 on thescreen. The display unit 42 may display a list of file names of datafiles storing individual data stored in the storage unit 40. The iconsor file names are used to specify data to be transmitted when the userdesires to transmit the data from the communication device 20 to anotherdevice.

The user I/F 44 provides an input means for allowing the user to inputinformation or to give an instruction. In this embodiment, the user I/F44 includes a pointing device, which enables the user to designate anyposition on the screen of the communication device 20. For example, thepointing device may be a touch panel configured integrally with thedisplay unit 42, or may be a mouse, a touch pad, or the like in placethereof. Further, the user I/F 44 may additionally include a keyboard, abutton, a wheel, or the like.

The communication I/F 46 intermediates communication with other devices(including the communication control device 100 and anothercommunication device 20) by the communication device 20. For example,the communication I/F 46 may be a wireless communication interface suchas a wireless local area network (LAN), Bluetooth (registeredtrademark), or WiMax (registered trademark), or may be a wiredcommunication interface such as a wired LAN or a universal serial bus(USB).

The control unit 48 controls an entire operation of the communicationdevice 20 by using a processor such as a central processing unit (CPU)or a digital signal processor (DSP). For example, the control unit 48operates unique functions of the communication device 20 (an informationprocessing function of a PC, a communication function of a smart phone,or other application functions). For example, the control unit 48 causesthe display unit 41 to display icons and the like respectivelycorresponding to data stored in the storage unit 40. In this embodiment,for example, when a predetermined user input related to datatransmission via the user I/F 44 is sensed, the control unit 48transmits a user input signal of notification of content of the userinput to the communication control device 100 via the communication I/F46. The user input related to the data transmission will be specificallydescribed later.

The control unit 48 reads data designated by the user from the storageunit 40 according to an instruction from the communication controldevice 100, and transmits the data to another communication device viathe communication I/F 46. For example, the transmission of data from thecommunication device 20 can be realized by a structure of file transferprotocol (FTP), hyper-text transfer protocol (HTTP), Samba, or the like.A data transmission destination is selected by the communication controldevice 100.

The communication control device 100 is a device for controlling datatransmission from the communication device 20 to another communicationdevice. The configuration of the communication control device 100according to this embodiment will be specifically described in the nextsection.

3. CONFIGURATION OF COMMUNICATION CONTROL DEVICE ACCORDING TO EMBODIMENT

FIG. 4 is a block diagram showing an example of a configuration of thecommunication control device 100 according to an embodiment of thepresent invention. Referring to FIG. 4, the communication control device100 includes an imaging unit 102, an environment map generating unit110, a communication processing unit 180, and a display control unit190.

[3-1. Imaging Unit]

For example, the imaging unit 102 may be realized as a camera having animaging element such as a charge coupled device (CCD) or a complementarymetal oxide semiconductor (CMOS). The imaging unit 102 may be providedoutside the communication control device 100. The imaging unit 102outputs an image generated by imaging the real space such as theenvironment 1 illustrated in FIG. 1 to the environment map generatingunit 110 and the display control unit 190 as the input image.

[3-2. Environment Map Generating Unit]

The environment map generating unit 110 generates an environment map,which represents positions of one or more physical objects present inthe real space and the like based on the input image input from theimaging unit 102 and feature data of an object to be described laterstored by a data storage unit 130. In this embodiment as illustrated inFIG. 4, the environment map generating unit 110 includes a self-positiondetecting unit 120, the data storage unit 130, an image recognizing unit140, an environment map building unit 150, and an environment mapstorage unit 152.

(1) Self-Position Detecting Unit

The self-position detecting unit 120 dynamically detects a position ofthe camera showing the input image on the basis of the input image inputfrom the imaging unit 102 and the feature data stored in the datastorage unit 130. For example, even when the imaging device has amonocular camera, the self-position detecting unit 120 may dynamicallydetermine a position and a posture of the camera and a position of an FPon an imaging plane of the camera for each frame by applying the SLAMtechnology disclosed in the above-described “Real-Time SimultaneousLocalization and Mapping with a Single Camera” (Andrew J. Davison,Proceedings of the 9th IEEE International Conference on Computer VisionVolume 2, 2003, pp. 1403-1410).

First, the entire flow of a self-position detection process by theself-position detecting unit 120 to which the SLAM technology is appliedwill be described with reference to FIG. 5. Next, the self-positiondetection process will be described in detail with reference to FIGS. 6to 8.

FIG. 5 is a flowchart illustrating an example of the flow of theself-position detection process by the self-position detecting unit 120to which the SLAM technology is applied. In FIG. 5, when theself-position detection process starts, the self-position detecting unit120 first initializes a state variable (step S102). In this embodiment,the state variable is a vector including a position and a posture(rotation angle) of the camera, a moving speed and an angular speed ofthe camera and positions of one or more FPs as elements. Theself-position detecting unit 120 then sequentially obtains the inputimages from the imaging unit 102 (step S112). The processes from stepS112 to step S118 may be repeated for each input image (that is, eachframe).

In step S114, the self-position detecting unit 120 tracks FPs present inthe input image. For example, the self-position detecting unit 120detects a patch (for example, a small image of 3×3=9 pixels having thecenter of an FP) of each FP stored in advance in the data storage unit130 from the input image. Here, the position of the detected patch, thatis, the position of the FP_(i) is used later when the state variable isupdated.

In step S116, the self-position detecting unit 120 generates a predictedvalue of the state variable after 1 frame, for example, based on apredetermined prediction model. Also, in step S118, the self-positiondetecting unit 120 updates the state variable by using the predictedvalue of the state variable generated in step S116 and an observed valuecorresponding to the position of the FP detected in step S114. Theself-position detecting unit 120 executes the processes of steps S116and S118 based on a principle of an extended Kalman filter.

As a result of such processing, a value of the state variable updatedfor each frame is output. Content of processes of tracking of the FP(step S114), prediction of the state variable (step S116), and updatingof the state variable (step S118) will be described more specifically.

(1-1) Tracking of Feature Point

In this embodiment, the data storage unit 130 stores in advance featuredata representing features of objects corresponding to physical objects(the communication device 20 and other physical objects), which may bepresent in the real space. For example, the feature data includes smallimages, that is, patches regarding one or more FPs, each representing afeature of the appearance of each object. For example, the patch may bethe small image including 3×3=9 pixels having the center of the FP.

FIG. 6 illustrates two examples of the objects and an example of FPs andpatches set on each object. A left object in FIG. 6 is an objectrepresenting a PC (see FIG. 6 a). A plurality of FPs including a featurepoint FP1 are set on the object. Further, a patch Pth1 is defined inassociation with the feature point FP1. On the other hand, a rightobject in FIG. 6 is an object representing a calendar (see FIG. 6 b). Aplurality of FPs including a feature point FP2 are set on the object.Further, a patch Pth2 is defined in association with the feature pointFP2.

Upon acquisition of an input image from the imaging unit 102, theself-position detecting unit 120 collates partial images included in theinput image and the patch for each FP illustrated in FIG. 6 stored inadvance in the data storage unit 130. The self-position detecting unit120 then specifies a position of each FP included in the input image(for example, a position of a center pixel of the detected patch) as aresult of collation.

For tracking FPs (step S114 in FIG. 5), it is not necessary to store inadvance data regarding all of the FPs to be tracked in the data storageunit 130. For example, four FPs are detected in the input image at timeT=t−1 in an example illustrated in FIG. 7 (see FIG. 7 a). Next, if theposition or the posture of the camera changes at time T=t, only two ofthe four FPs shown in the input image at time T=t−1 are shown in theinput image. In this case, the self-position detecting unit 120 maynewly set FPs in positions having a feature pixel pattern of the inputimage and may use the new FPs in the self-position detection process fora subsequent frame. For example, in the example of FIG. 7, three new FPsare set on the object at time T=t (see FIG. 7 b). This is one feature ofthe SLAM technology. Thereby, the cost of presetting all FPs can bereduced and the accuracy of processing can be improved using a number ofFPs to be tracked.

(1-2) Prediction of State Variable

In this embodiment, the self-position detecting unit 120 uses a statevariable X shown in the following Equation as the state variable towhich the extended Kalman filter to be is applied.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack & \; \\{X = \begin{pmatrix}x \\\omega \\\overset{.}{x} \\\overset{.}{\omega} \\p_{1} \\\vdots \\p_{N}\end{pmatrix}} & (1)\end{matrix}$

The first element of the state variable X in Equation (1) represents athree-dimensional position of the camera in a global coordinate system(x, y, z) being a coordinate system set in the real space, as expressedin the following Equation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack & \; \\{x = \begin{pmatrix}x_{c} \\y_{c} \\z_{c}\end{pmatrix}} & (2)\end{matrix}$

Also, the second element of the state variable is a four-dimensionalvector co having a quaternion as an element corresponding to a rotationmatrix representing the posture of the camera. The posture of the cameramay be represented using an Euler angle in place of the quaternion.Also, the third and fourth elements of the state variable represent themoving speed and the angular speed of the camera, respectively.

Further, the fifth and subsequent elements of the state variablerepresent a three-dimensional position p_(i) of a feature point FP_(i)(i=1 . . . N) in the global coordinate system as expressed in thefollowing Equation. As described above, the number of the FPs, N, maychange during processing.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack & \; \\{p_{i} = \begin{pmatrix}x_{i} \\y_{i} \\z_{i}\end{pmatrix}} & (3)\end{matrix}$

The self-position detecting unit 120 generates a predicted value of thestate variable in a latest frame based on the value of the statevariable X initialized in step S102 or the value of the state variable Xupdated in a previous frame. The predicted value of the state variableis generated according to a state equation of the extended Kalman filteraccording to multidimensional normal distribution as shown in thefollowing Equation.

[Equation 4]

predicted state variable {circumflex over (X)}=F(X,a)+w  (4)

Here, F represents a prediction model regarding the state transition ofa system and a represents a prediction condition. Also, w representsGaussian noise, and, for example, may include a model approximationerror, an observation error, and the like. In general, an average of theGaussian noise w is 0.

FIG. 8 is an illustrative diagram illustrating an example of theprediction model according to this embodiment. Referring to FIG. 8, twoprediction conditions in the prediction model according to thisembodiment are shown. First, as a first condition, it is assumed thatthe three-dimensional position of the FP in the global coordinate systemdoes not change. That is, if the three-dimensional position of thefeature point FP1 at time T is p_(T), the following relationship isestablished.

[Equation 5]

P_(t)=p_(t-1)  (5)

Next, as a second condition, it is assumed that the motion of the camerais uniform motion. That is, the following relationship is establishedfor the speed and the angular speed of the camera from time T=t−1 totime T=t.

[Equations 6]

{dot over (x)}_(t)={dot over (x)}_(t-1)  (6)

{dot over (ω)}_(t)={dot over (ω)}_(t-1)  (7)

The self-position detecting unit 120 generates a predicted value of thestate variable for the latest frame based on the prediction model andthe state equation expressed in Equation (4).

(1-3) Updating of State Variable

For example, the self-position detecting unit 120 evaluates an errorbetween observation information predicted from the predicted value ofthe state variable and actual observation information obtained as aresult of FP tracking, using an observation equation. In Equation (8), vis the error.

[Equations 7]

observation information s=H({circumflex over (X)})+v  (8)

predicted observation information ŝ=H({circumflex over (X)})  (9)

Here, H represents an observation model. For example, a position of thefeature point FP_(i) on the imaging plane (u-v plane) is defined asexpressed in the following Equation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 8} \right\rbrack & \; \\{{{position}\mspace{14mu} {of}\mspace{14mu} {FP}_{i}\mspace{14mu} {on}\mspace{14mu} {imaging}\mspace{14mu} {plane}\mspace{14mu} {\overset{\sim}{p}}_{i}} = \begin{pmatrix}u_{i} \\v_{i} \\1\end{pmatrix}} & (10)\end{matrix}$

Here, all of the position x of the camera, the posture co of the camera,and the three-dimensional position p_(i) of the feature point FP_(i) aregiven as the elements of the state variable X. Then, the position of thefeature point FP_(i) on the imaging plane is derived using the followingEquation according to a pinhole model.

[Equation 9]

λ{tilde over (p)} _(i) =AR _(ω)(p _(i) −x)  (11)

Here, λ represents a parameter for normalization, A represents a camerainternal parameter, and R_(ω) represents the rotation matrixcorresponding to the quaternion ω representing the posture of the cameraincluded in the state variable X. The camera internal parameter A isgiven in advance as expressed in the following Equation according tocharacteristics of the imaging device, which takes the input image.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 10} \right\rbrack & \; \\{A = \begin{pmatrix}{{- f} \cdot k_{u}} & {{f \cdot k_{u} \cdot \cot}\; \theta} & u_{O} \\0 & {- \frac{f \cdot k_{v}}{\sin \; \theta}} & v_{O} \\0 & 0 & 1\end{pmatrix}} & (12)\end{matrix}$

Here, f represents a focal distance, θ represents orthogonality of animage axis (ideal value is 90 degrees), k_(u) represents a scale of avertical axis of the imaging plane (a scale change rate from the globalcoordinate system to the coordinate system of the imaging plane), k_(v)represents a scale of a horizontal axis of the imaging plane, and(u_(o), v_(o)) represents a center position of the imaging plane.

Therefore, a feasible latest state variable X may be obtained bysearching for the state variable X_(s) which minimizes the error betweenthe predicted observation information derived using Equation (11), thatis, the position of each FP on the imaging plane and the result of FPtracking in step S114 of FIG. 5.

[Equation 11]

latest state variable X→{circumflex over (X)}+Innov(s−ŝ)  (13)

The self-position detecting unit 120 outputs the position x and theposture co of the camera (imaging device) dynamically updated byapplying the SLAM technology to the environment map building unit 150,the communication processing unit 180, and the display control unit 190.

(2) Data Storage Unit The data storage unit 130 stores in advancefeature data representing features of an object corresponding to aphysical object, which may be present in the real space, by using astorage medium such as a hard disk or a semiconductor memory. An examplein which the data storage unit 130 is part of the environment mapgenerating unit 110 is illustrated in FIG. 4, but the present inventionis not limited to such an example, and the data storage unit 130 may beprovided outside the environment map generating unit 110. FIG. 9 is anillustrative diagram illustrating an example of the configuration of thefeature data.

Referring to FIG. 9, feature data FD1 is shown as an example in anobject Obj1. The feature data FD1 includes an object name FD11, imagedata FD12 taken from six directions, patch data FD13, three-dimensionalshape data FD14, and ontology data FD15.

The object name FD11 is a name by which a corresponding object can bespecified such as “Smart Phone A.”

For example, the image data FD12 includes six image data obtained bytaking images of the corresponding object from six directions (front,back, left, right, above, and below). The patch data FD13 is a set ofsmall images having the center of each FP for each of one or more FPsset on each object. The image data FD12 and the patch data FD13 may beused for an object recognition process by the image recognizing unit 140to be described later. Also, the patch data FD13 may be used for theabove-described self-position detection process by the self-positiondetecting unit 120.

The three-dimensional shape data FD14 includes polygon information forrecognizing a shape of the corresponding object and three-dimensionalposition information of FPs. The three-dimensional shape data FD14 maybe used for an environment map building process by the environment mapbuilding unit 150 to be described later.

For example, the ontology data FD15 is the data, which may be used toassist in the environment map building process by the environment mapbuilding unit 150. In the example of FIG. 9, the ontology data FD15indicates that the object Obj1, which is the smart phone, is more likelyto come in contact with an object corresponding to a table and is lesslikely to come in contact with an object corresponding to a bookshelf.

(3) Image Recognizing Unit

The image recognizing unit 140 specifies which object corresponds toeach of physical objects shown in the input image by using theabove-described feature data stored in the data storage unit 130.

FIG. 10 is a flowchart illustrating an example of a flow of the objectrecognition process by the image recognizing unit 140. Referring to FIG.10, the image recognizing unit 140 first obtains an input image from theimaging unit 102 (step S212). Next, the image recognizing unit 140collates partial images included in the input image and patches of oneor more FPs of each object included in feature data, and extracts FPsincluded in the input image (step S214). The FPs used in the objectrecognition process by the image recognizing unit 140 may notnecessarily be the same as the FPs used in the self-position detectionprocess by the self-position detecting unit 120. However, if common FPsare used in both the processes, the image recognizing unit 140 may reusea tracking result of FPs by the self-position detecting unit 120.

Next, the image recognizing unit 140 specifies the object shown in theinput image based on an extraction result of FPs (step S216). Forexample, if the FPs belonging to one object are extracted with highdensity in a certain area, the image recognizing unit 140 may recognizethat the object is shown in the area. The image recognizing unit 140then outputs an object name (or an identifier) of the specified objectand positions of the FPs belonging to the object on the imaging plane tothe environment map building unit 150 (step S218).

(4) Environment Map Building Unit

The environment map building unit 150 generates an environment map byusing the position and the posture of the camera input from theself-position detecting unit 120, the positions of the FPs on theimaging plane input from the image recognizing unit 140 and the featuredata stored in the data storage unit 130. In this specification, theenvironment map is a set of data representing positions (and postures)of one or more physical objects present in the real space. Specifically,in this embodiment, the environment map includes data representingpositions and postures of two or more communication devices 20 asphysical objects present in the real space. Also, for example, theenvironment map may include object names corresponding to physicalobjects, three-dimensional positions of FPs belonging to the physicalobjects, polygon information configuring shapes of the physical objects,and the like. For example, the environment map may be built by obtainingthe three-dimensional position of each FP according to theabove-described pinhole model from the positions of the FPs on theimaging plane input from the image recognizing unit 140.

If the relation equation of the pinhole model expressed in Equation (11)is modified, the three-dimensional position p_(i) of the feature pointFP_(i) in the global coordinate system may be obtained by the followingEquation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 12} \right\rbrack & \; \\{p_{i} = {{x + {\lambda \cdot R_{\omega}^{T} \cdot A^{- 1} \cdot {\overset{\sim}{p}}_{i}}} = {x + {{d \cdot R_{\omega}^{T}}\; \frac{A^{- 1} \cdot {\overset{\sim}{p}}_{i}}{{A^{- 1} \cdot {\overset{\sim}{p}}_{i}}}}}}} & (14)\end{matrix}$

Here, d represents a distance between the camera and each FP in theglobal coordinate system. The environment map building unit 150 maycalculate the distance d based on the positions of at least four FPs onthe imaging plane and the distance between the FPs for each object. Thedistance between the FPs is stored in advance in the data storage unit130 as the three-dimensional shape data FD14 included in the featuredata described with reference to FIG. 9. A process of calculating thedistance d in Equation (14) is disclosed in detail in theabove-described Japanese Patent Laid-Open Publication No. 2008-304268.

If the distance d is calculated, the remaining variables of a right sideof Equation (14) are the position and the posture of the camera inputfrom the self-position detecting unit 120 and the position of the FP onthe imaging plane input from the image recognizing unit 140, and all ofwhich are known. The environment map building unit 150 then calculatesthe three-dimensional position in the global coordinate system for eachFP input from the image recognizing unit 140 according to Equation (14).The environment map building unit 150 then builds a latest environmentmap according to the calculated three-dimensional position of each FPand causes the environment map storage unit 152 to store the builtenvironment map. At this time, the environment map building unit 150 mayimprove the accuracy of data of the environment map by using theontology data FD15 included in the feature data described with referenceto FIG. 9.

The environment map storage unit 152 stores the environment map built bythe environment map building unit 150 by using the storage medium suchas the hard disk or the semiconductor memory. In this embodiment, in theenvironment map storage unit 152, data related to each communicationdevice 20 is associated with identification information of eachcommunication device 20 registered in advance (for example, an IPaddress, a host name, a MAC address, or the like).

[3-3. Communication Processing Unit]

The communication processing unit 180 controls data transmission fromone communication device 20 to another device according to a user inputreceived from the communication device 20 by using an environment mapgenerated by the environment map generating unit 110. In this embodimentas shown in FIG. 4, the communication processing unit 180 includes acommunication I/F 182, a detecting unit 184, a selecting unit 186, and acommunication control unit 188.

(1) Communication I/F

The communication I/F 182 intermediates communication with thecommunication device 20 by the communication control device 100. Forexample, the communication I/F 182 may be the same wireless or wiredcommunication interface as the communication I/F 46 of the communicationdevice 20. The communication I/F 182 outputs a signal received from thecommunication device 20 to the detecting unit 184 and the communicationcontrol unit 188. Also, the communication I/F 182 transmits a signalinput from the communication control unit 188 to the communicationdevice 20.

(2) Detecting Unit

The detecting unit 184 detects a user input by a user toward thecommunication device 20, which designates any data provided in thecommunication device 20 and a direction, by monitoring the signalreceived by the communication I/F 182. For example, the user input,which designates any data provided in the communication device 20 andthe direction, is converted into a user input signal IN in thecommunication device 20, and is transmitted from the communicationdevice 20 to the communication control device 100. When detecting thatthe user input signal IN is received by the communication I/F 182, thedetecting unit 184 notifies the selecting unit 186 of content of theuser input, and causes the selecting unit 186 to select a device to bedesignated as a transmission destination of data from the communicationdevice 20 of a transmission source of the user input signal IN.

FIG. 11 is an illustrative diagram illustrating a user input to thecommunication device 20 to be detected by the detecting unit 184.Referring to FIG. 11, an input image Im1 is shown as an example to beimaged by the camera of the communication control device 100. Thecommunication devices 20 a, 20 b, and 20 c are shown in the input imageIm1. Two icons respectively representing data D1 and D2 are displayed onthe screen of the communication device 20 a.

In a situation as shown in FIG. 11, for example, the user who desires totransmit the data D2 to the communication device 20 b drags the iconrepresenting the data D2 in a direction (a right direction of thefigure) in which the communication device 20 b is present. Thecommunication device 20 a then senses the drag event and recognizes theicon designated by the user input and the drag direction. Thecommunication device 20 a transmits a user input signal IN havingcontent of information (for example, a data file path) for specifyingthe data D2 corresponding to the designated icon and the drag directionto the communication control device 100. The detecting unit 184 of thecommunication control device 100 monitors the signal received by thecommunication I/F 182, and detects the user input signal IN.

For example, the detecting unit 184 may detect the user input to thecommunication device 20 by monitoring a set of input images input fromthe imaging unit 102, and recognizing an image of an operation of theuser shown in the input images, instead of detecting the user inputsignal IN transmitted from the communication device 20.

(3) Selecting Unit

The selecting unit 186 selects a communication device, which is atransmission destination of the designated data from an environment mapgenerated by the environment map generating unit 110, on the basis ofthe direction designated by the user input. In this embodiment, forexample, the selecting unit 186 selects a communication device locatedin the direction designated by the user input with respect to thecommunication device 20 of the transmission source of the user inputsignal IN in the environment map as a device of the data transmissiondestination. For example, if there are a plurality of communicationdevices located in the direction, the selecting unit 186 can select adevice located closest to the communication device 20 of thetransmission source among the plurality of devices as the datatransmission destination device.

FIGS. 12 and 13 are illustrative diagrams illustrating the selection ofthe transmission destination by the selecting unit 186 according to thisembodiment.

Referring to FIG. 12, the communication devices 20 a, 20 b, and 20 cincluded in the environment map are shown. A position of thecommunication device 20 a is retained in the environment map as athree-dimensional position X_(s) in the global coordinate system (x, y,z). A posture of the communication device 20 a is retained in theenvironment map as a quaternion ω_(s) corresponding to a rotation matrixin the global coordinate system (x, y, z) of the environment map. Theposture ω_(s) of the communication device 20 a corresponds to a rotationangle of a normal line of the screen of the communication device 20 a.If a drag direction (u, v) on the screen of the communication device 20a included in the user input signal IN is defined in addition to theposition X_(s) and the posture ω_(s) of the communication device 20 a,the position X_(s) is designated as a start point and the user canuniquely specify a straight line L1 along the direction designated bythe drag in the global coordinate system of the environment map.

The selecting unit 186 specifies the straight line L1 as describedabove, and selects a communication device 20 located on the straightline L1 or located at least near the straight line L1 as the datatransmission destination device. In the example of FIG. 12, thecommunication device 20 b is located on the straight line L1 having theposition X_(s) of the communication device 20 a as the start point. Onthe other hand, the communication device 20 c is located away from thestraight line L1. Here, the selecting unit 186 selects the communicationdevice 20 b as the device of the transmission destination of data fromthe communication device 20 a.

Referring to FIG. 13, the communication devices 20 a, 20 b, and 20 cincluded in the environment map are shown. These devices have apositional relationship different from the example of FIG. 12. Even inthis case, the selecting unit 186 specifies the straight line L1 havingthe start point at the communication device 20 a on the basis of theposition X_(s) and the posture ω_(s) of the communication device 20 a inthe environment map and the drag direction (u, v) designated by the userinput. In the example of FIG. 13, the communication devices 20 b and 20c are all located on the straight line L1. At this time, for example,the selecting unit 186 can select the communication device 20 b, locatedcloser to the communication device 20 a, as a device of a transmissiondestination of data from the communication device 20 a, from thecommunication devices 20 b and 20 c. Alternatively, for example, theselecting unit 186 can display a message for designating any device (thecommunication device 20 b or 20 c) to the user, and can select anydevice as the data transmission destination device according to a resultof a designation by the user.

FIG. 14 is a flowchart showing an example of a flow of a transmissiondestination selection process by the selecting unit 186 according tothis embodiment.

Referring to FIG. 14, the selecting unit 186 first acquires a positionand a posture of each communication device from an environment map (stepS302). Next, the selecting unit 186 designates a position of acommunication device 20 of a transmission source as a start point, andspecifies a straight line L1 along a direction designated by a userinput (step S304).

Next, the selecting unit 186 determines whether or not anothercommunication is at least near the specified straight line L1 (that is,on the straight line L1 or near the straight line L1) (step S306). Forexample, it can be determined that a device having a distance from thestraight line L1, which is less than a preset threshold value, islocated near the straight line L1. Here, if no other communicationdevice is located near the straight line L1, the selecting unit 186outputs an error to the communication control unit 188 (step S314) andterminates the process. On the other hand, if another communicationdevice is present at least near the straight line L1, the processproceeds to step S308.

Next, the selecting unit 186 determines whether or not a plurality ofother communication devices are present at least near the straight lineL1 (step S308). Here, if the plurality of other communication devicesare present at least near the straight line L1, the selecting unit 186selects a communication device located nearest the communication device20 of the transmission source as a data transmission destination device(step S310). On the other hand, if only one other communication deviceis located at least near the straight line L1, the selecting unit 186selects the other communication device as the data transmissiondestination device (step S312).

The selecting unit 186 outputs identification information foridentifying the data transmission destination device selected by theabove-described transmission destination selection process to thecommunication control unit 188. The data transmission destination deviceselected by the selecting unit 186 may not necessarily have theequivalent configuration of the communication device 20 according tothis embodiment. For example, if a general communication device, whichdoes not have a function of transmitting the user input signal IN to thecommunication control device 100, can be recognized in the environmentmap generated by the environment map generating unit 110, the generalcommunication device can be selected as the data transmissiondestination device by the selecting unit 186.

(4) Communication Control Unit

If a user input is detected by the detecting unit 184, the communicationcontrol unit 188 transmits data designated by the user input from acommunication device 20 serving as a target of the user input to anothercommunication device. More specifically, for example, if a user inputsignal IN is received from one communication device 20, thecommunication control unit 188 transmits an instruction signal SIGthrough which a data transmission instruction is given to the onecommunication device 20 via the communication I/F 182. The instructionsignal SIG is a signal for instructing to transmit the data designatedby the user input to another communication device selected by theselecting unit 186 according to the above-described transmissiondestination selection process. As a result, the data designated by theuser input is transmitted from the one communication device 20 toanother communication device located in a direction designated by theuser input.

The communication control unit 188 transmits the instruction signal SIGand also outputs identification information for identifying the datatransmission source device and the data transmission destination deviceto the display control unit 190.

[3-4. Display Control Unit]

The display control unit 190 displays an animation indicating datatransmission on a screen of a display device when data is transmittedfrom a data transmission source device to a data transmissiondestination device. For example, the display control unit 190 generatesan output image by superimposing the animation indicating the datatransmission on an input image input from the imaging unit 102. Thedisplay control unit 190 displays the generated output image on the HMD104. For example, the animation indicating the data transmission may bean animation in which any virtual object (a figure, a symbol, or acharacter) moves from the data transmission source device to the datatransmission destination device.

More specifically, for example, the display control unit 190 calculatesa position where the animation is superimposed on the input imageaccording to Equation (11) of the pinhole model by using a position anda posture of the camera acquired from the environment map generatingunit 110. For example, a display position of a start point of theanimation is calculated by substituting a three-dimensional position ofthe data transmission source device into the three-dimensional positionp_(i) of the feature point FP_(i) of the right side in Equation (11).Also, a display position of an end point of the animation is calculatedby substituting a three-dimensional position of the data transmissiondestination device into the three-dimensional position p_(i) of thefeature point FP_(i) of the right side in Equation (11). The displaycontrol unit 190 superimposes the animation in which the object movesbetween the display position of the start point and the display positionof the end point over a set of input images.

FIG. 15 is an illustrative diagram showing an example of an animationrelated to data transmission according to this embodiment. Referring toFIG. 15, output images Im2 a to Im2 c on which a set of animations 192 ato 193 a is superimposed are shown in one illustrative diagram forconvenience. Actually, the animation 192 a may be superimposed on theoutput image Im2 a, the animation 192 b on the output image Im2 b, andthe animation 192 c on the output image Im2 c in order along a timeaxis. The animations shown in FIG. 15 have shapes of arrows. Asdescribed above, the animations 192 a to 192 c are sequentiallydisplayed, so that the user can know that data is transmitted from thecommunication device 20 a to the communication device 20 d. The datatransmission destination device may not necessarily be shown in theinput image or the output image as long as its position is recognized inthe environment map. That is, data may be transmitted to a devicelocated outside a range of the screen among communication deviceslocated in the direction designated by the user input.

[3.5. Flow of Communication Control Process]

FIG. 16 is a flowchart showing an example of a flow of the communicationcontrol process by the communication control device 100 according tothis embodiment.

Referring to FIG. 16, first, the detecting unit 184 monitors thepresence/absence of a predetermined user input to the communicationdevice 20 (step S352). For example, if the detecting unit 184 detects auser input (a drag or the like) designating any data icon and adirection for the communication device 20, the process proceeds to stepS354.

Next, the selecting unit 186 acquires an environment map generated bythe environment map generating unit 110 (step S354). Here, the acquiredenvironment map includes an object corresponding to the communicationdevice 20 serving as a target of the user input detected by thedetecting unit 184 and an object corresponding to another communicationdevice.

Next, the transmission destination selection process described withreference to FIG. 14 is performed by the selecting unit 186 (step S356).Thereby, a device of a transmission destination of data from thecommunication device 20 serving as the target of the user input isselected.

Next, the communication control unit 188 determines the presence/absenceof the data transmission destination device in a selection result (stepS358). For example, if no communication device is present in thedirection designated by the user input, an error is output from theselecting unit 186 to the communication control unit 188. In this case,the communication control unit 188 instructs the display control unit190 to display an error message on the screen (step S360).

On the other hand, if the data transmission destination device ispresent in the selection result by the selecting unit 186, thecommunication control unit 188 transmits an instruction signal SIG tothe communication device serving as the target of the user input, andtransmits data to the data transmission destination device selected bythe selecting unit 186 (step S362). During the data transmission, thedisplay control unit 190 displays an animation indicating the datatransmission on the screen of the HMD 104 (step S364).

4. MODIFIED EXAMPLES

In the above-described embodiment, the selecting unit 186 of thecommunication control device 100 selects a device serving as a datatransmission destination from the environment map on the basis of thedirection designated by the user input of the drag or the like. Thereby,the user can intuitively designate the transmission destination upondata transmission between the communication devices. On the other hand,the intuitive designation of the transmission destination can beperformed according to other methods. In this section, other methods,which make it possible to perform the intuitive designation of thetransmission destination, will be described as modified examples of theabove-described embodiment.

FIG. 17 is an illustrative diagram illustrating the selection of thetransmission destination by the selecting unit 186 according to amodified example.

Referring to FIG. 17, the communication devices 20 a, 20 b, and 20 cincluded in the environment map are shown. Two icons respectivelyrepresenting data D1 and D2 are displayed on the screen of thecommunication device 20 a. In a situation as shown in FIG. 17, forexample, the user who desires to transmit the data D2 to thecommunication device 20 b holds the communication device 20 a above thecommunication device 20 b in reality, and clicks or taps the iconrepresenting the data D2. Then, the communication device 20 a senses theuser input and transmits a user input signal IN to the communicationcontrol device 100. The detecting unit 184 of the communication controldevice 100 detects the user input signal IN.

At this time, a position of the communication device 20 a is retained inthe environment map as a three-dimensional position X_(s) in the globalcoordinate system (x, y, z) of the environment map. Here, the selectingunit 186 designates the position X_(s) of the communication device 20 aas a start point, and specifies a straight line L2 extending downwardalong a vertical line in the global coordinate system of the environmentmap. The selecting unit 186 selects a communication device 20 located onthe straight line L2 or located at least near the straight line L2 asthe data transmission destination device. In the example of FIG. 17, thecommunication device 20 b is located on the straight line L2. On theother hand, the communication device 20 c is located away from thestraight line L2. Here, the selecting unit 186 selects the communicationdevice 20 b as the device of the transmission destination of data fromthe communication device 20 a.

FIG. 18 is a flowchart showing an example of a flow of the transmissiondestination selection process by the selecting unit 186 according to amodified example.

Referring to FIG. 18, the selecting unit 186 first acquires a positionof each communication device from the environment map (step S402). Next,the selecting unit 186 designates a position of a communication device20 of a transmission source as a start point, and specifies a straightline L2 extending downward along a vertical line (step S404).

Next, the selecting unit 186 determines whether or not anothercommunication device is present at least near the straight line L2 (thatis, on the straight line L2 or near the straight line L2) (step S406).For example, it can be determined that a device having a distance fromthe straight line L2, which is less than a preset threshold value, islocated near the straight line L2. Here, if no other communicationdevice is located near the straight line L2, the selecting unit 186outputs an error to the communication control unit 188 (step S414) andterminates the process. On the other hand, if another communicationdevice is present at least near the straight line L2, the processproceeds to step S408.

Next, the selecting unit 186 determines whether or not a plurality ofother communication devices are present at least near the straight lineL2 (step S408). Here, if the plurality of other communication devicesare present at least near the straight line L2, the selecting unit 186selects a communication device located nearest the communication device20 of the transmission source as a data transmission destination device(step S410). On the other hand, if only one other communication deviceis located at least near the straight line L2, the selecting unit 186selects the other communication device as the data transmissiondestination device (step S412).

According to the transmission destination selection process related tothe modified example, the user can intuitively transmit data provided inthe communication device to another communication device as if the datawere dropped due to gravity.

5. SUMMARY

Up to now, the communication devices 20 and the communication controldevice 100 according to the embodiments of the present invention and themodified examples of the communication control device 100 have beendescribed in detail with reference to FIGS. 1 to 18. According to thecommunication control device 100, an environment map including aplurality of objects corresponding to a plurality of communicationdevices 20 is dynamically built on the basis of input images obtained byimaging a real space. If a user input to a communication device 20 of adata transmission source is detected, designated data is transmitted toa data transmission destination device selected from the environment mapon the basis of a direction designated by the user input. Alternatively,the data transmission destination device can be selected on the basis ofa direction of gravity work in place of a direction designated by a userinput. Thereby, the user can intuitively perform various types of datatransmission from a PC to a printer, from a digital camera to a PC, andfrom a mobile terminal to a PC by a simple user input (for example,simple and easy manipulation such as a drag, a click, or the like) tothe transmission source device.

A device selected as the data transmission destination may be a devicelocated in the direction designated by the user input or the directionin which gravity works with respect to the data transmission sourcedevice in the environment map. The communication control device 100selects the data transmission destination device on the basis ofinformation such as a position of the data transmission source device inthe environment map. That is, it is possible to provide an intuitiveuser interface corresponding to a positional relationship ofcommunication devices by utilizing selection of data transmissiondestination device with application of the environment map. At thistime, if a plurality of communication devices are present in thedesignated direction or the direction in which gravity works, thecommunication control device 100 can select a device located nearest thedata transmission source device as the data transmission destination.That is, the data transmission destination can be uniquely selected evenin an environment where the plurality of communication devices arepresent by applying a corresponding environment map to a real space.

The communication control device 100 may cause the display device todisplay an animation representing data transmission when data istransmitted from the device of the data transmission source to thedevice of the data transmission destination. The communication controldevice 100 can enable the user to easily know devices to/from which datais transmitted by displaying the animation on the basis of positions ofthe data transmission source device and the data transmissiondestination device in the environment map.

In this specification, an example in which the communication device 20and the communication control device 100 are physically configured inseparate bodies has been mainly described. However, various functions ofthe communication control device 100 excluding the camera and the HMDmay be mounted in the physically same device as the communication device20 of the data transmission source.

The set of processes by the communication device 20 and thecommunication control device 100 described in this specification istypically realized using software. For example, a program configuringthe software, which realizes the set of processes, is stored in advancein a storage medium inside or outside each device. For example, eachprogram is read to a random access memory (RAM) of each device uponexecution, and is executed by a processor such as a CPU or the like.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2010-022973 filedin the Japan Patent Office on Feb. 4, 2010, the entire content of whichis hereby incorporated by reference.

1. A communication control device comprising: a data storage unit storing feature data representing features of appearances of one or more communication devices; an environment map building unit for building an environment map representing positions of communication devices present in a real space on the basis of an input image obtained by imaging the real space and the feature data stored in the data storage unit; a detecting unit for detecting a user input toward a first communication device designating any data provided in the first communication device and a direction; a selecting unit for selecting a second communication device serving as a transmission destination of the designated data from the environment map on the basis of the direction designated by the user input; and a communication control unit for transmitting the data provided in the first communication device from the first communication device to the second communication device.
 2. The communication control device according to claim 1, wherein the selecting unit selects a communication device located in the direction designated by the user input with respect to the first communication device in the environment map as the second communication device.
 3. The communication control device according to claim 2, wherein the selecting unit specifies one straight line having a start point at the first communication device on the basis of a position and a posture of the first communication device in the environment map and the direction designated by the user input, and selects a communication device located at least near the straight line as the second communication device.
 4. The communication control device according to claim 2, wherein if a plurality of communication devices are located in the direction designated by the user input with respect to the first communication device in the environment map, the selecting unit selects a communication device located nearest the first communication device among the plurality of communication devices as the second communication device.
 5. The communication control device according to claim 1, further comprising a display control unit for causing a display device to display an animation representing data transmission when the data is transmitted from the first communication device to the second communication device.
 6. The communication control device according to claim 1, wherein the user input is dragging an icon displayed on a screen provided in the first communication device and the data to be transmitted from the first communication device is specified according to which icon is dragged by a user.
 7. A communication control method by a communication control device including a storage medium storing feature data representing features of appearances of one or more communication devices, comprising the steps of: building an environment map representing positions of communication devices present in a real space on the basis of an input image obtained by imaging the real space and the feature data; detecting a user input toward a first communication device designating any data provided in the first communication device and a direction; selecting a second communication device serving as a transmission destination of the designated data from the environment map on the basis of the direction designated by the user input; and transmitting the data provided in the first communication device from the first communication device to the second communication device.
 8. A program for causing a computer, which controls a communication control device including a storage medium storing feature data representing features of appearances of one or more communication devices, to function as: an environment map building unit for building an environment map representing positions of communication devices present in a real space on the basis of an input image obtained by imaging the real space and the feature data stored in the data storage unit; a detecting unit for detecting a user input toward a first communication device designating any data provided in the first communication device and a direction; a selecting unit for selecting a second communication device serving as a transmission destination of the designated data from the environment map on the basis of the direction designated by the user input; and a communication control unit for transmitting the data provided in the first communication device from the first communication device to the second communication device.
 9. A communication control device comprising: a data storage unit storing feature data representing features of appearances of one or more communication devices; an environment map building unit for building an environment map representing positions of communication devices present in a real space on the basis of an input image obtained by imaging the real space and the feature data stored in the data storage unit; a detecting unit for detecting a user input toward a first communication device designating any data provided in the first communication device; a selecting unit for selecting a second communication device serving as a transmission destination of the designated data from the environment map on the basis of a position of the first communication device in the environment map; and a communication control unit for transmitting the data provided in the first communication device from the first communication device to the second communication device.
 10. The communication control device according to claim 9, wherein the selecting unit selects a communication device located below the first communication device in the environment map as the second communication device.
 11. A communication control method by a communication control device including a storage medium storing feature data representing features of appearances of one or more communication devices, comprising the steps of: building an environment map representing positions of communication devices present in a real space on the basis of an input image obtained by imaging the real space and the feature data; detecting a user input toward a first communication device designating any data provided in the first communication device; selecting a second communication device serving as a transmission destination of the designated data from the environment map on the basis of a position of the first communication device in the environment map; and transmitting the data provided in the first communication device from the first communication device to the second communication device.
 12. A program for causing a computer, which controls a communication control device including a storage medium storing feature data representing features of appearances of one or more communication devices, to function as: an environment map building unit for building an environment map representing positions of communication devices present in a real space on the basis of an input image obtained by imaging the real space and the feature data stored in the data storage unit; a detecting unit for detecting a user input toward a first communication device designating any data provided in a first communication device; a selecting unit for selecting a second communication device serving as a transmission destination of the designated data from the environment map on the basis of a position of the first communication device in the environment map; and a communication control unit for transmitting the data provided in the first communication device from the first communication device to the second communication device. 